Subjective and Objective Evaluation of Conversational Agents
نویسندگان
چکیده
In this paper we present results from an investigation on correlations between subjective and objective evaluation metrics for young people using a conversational agent. The subjective evaluation metrics capture users’ experiences of different aspects of conversations with a virtual agent while the objective evaluation metrics are based on an analysis of the actual conversation between the users and the agent. Our study has been conducted using a conversational agent incorporated in a learning environment. The users in the study were pupils in a regular school aged 12 to 14 years. Our results show that there are no correlations between subjective and objective metrics that are supposed to measure the same aspects, for example, to what extent the system can correctly interpret and give appropriate responses to user utterances. However, users that subjectively like the conversational agent rate its conversational behaviour higher than those that dislikes the system, even though there is no corresponding difference for the objective measures.
منابع مشابه
Subjective and Objective Evaluation of Conversational Agents in Learning Environments for Young Teenagers
In this paper we present results from a study of subjective and objective evaluation metrics used to asses a conversational agent. Our study has been conducted in a school setting with students, aged 12 to 14 years old, who used a virtual learning environment that incorporates social conversation with a pedagogical agent. The subjective evaluation metrics capture the students’ experiences of di...
متن کاملOn the Evaluation of the Conversational Speech Quality in Telecommunications
We propose an objective method to assess speech quality in the conversational context by taking into account the talking and listening speech qualities and the impact of delay. This approach is applied to the results of four subjective tests on the effects of echo, delay, packet loss, and noise. The dataset is divided into training and validation sets. For the training set, a multiple linear re...
متن کاملExtended ratio edge detector for despeckled SAR image evaluation
Synthetic aperture radar (SAR) images due to the usage of coherent imaging systems are affected by speckle. So lots of despeckling filters have been introduced up to now to suppress the speckle. Hence, objective and subjective evaluation of the denoised SAR images becomes a necessity. Thereby lots of objective evaluating estimators are introduced to evaluate the performance of despeckling filte...
متن کاملA Survey on Evaluation Metrics for Backchannel Prediction Models
In this paper we give an overview of the evaluation metrics used to measure the performance of backchannel prediction models. Both objective and subjective evaluation metrics are discussed. The survey shows that almost every backchannel prediction model is evaluated with a different evaluation metric. This makes comparison between developed models unreliable, even beside the other variables in ...
متن کاملThe Relationship between Subjective Evaluation of Stressors and Depression in Menopausal Women: The Mediating Role of Life Satisfaction
Objective: Previous studies have shown that menopausal women are more likely to experience depression. However, there are few studies that investigated the cognitive mechanism that may have a role in developing depression in menopausal women. Thus, the present study aimed to investigate the mediating role of life satisfaction in the relation between subjective evaluation of stressors and depres...
متن کامل